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Abstract 

We present a new view of Gaussian belief 
propagation (GaBP) based on a representa- 
tion of the determinant as a product over or- 
bits of a graph. We show that the GaBP 
determinant estimate captures totally back- 
tracking orbits of the graph and consider how 
to correct this estimate. We show that the 
missing orbits may be grouped into equiva- 
lence classes corresponding to backtrackless 
orbits and the contribution of each equiv- 
alence class is easily determined from the 
GaBP solution. Furthermore, we demon- 
strate that this multiplicative correction fac- 
tor can be interpreted as the determinant of a 
backtrackless adjacency matrix of the graph 
with edge weights based on GaBP. Finally, 
an efficient method is proposed to compute 
a truncated correction factor including all 
backtrackless orbits up to a specified length. 



1. Introduction 

Belief Propagation is a widely used method for infer- 
ence in graphical models. We study this algorithm 
in the context of Gaussian graphical models. There 
have been several studies of Gaussian belief propa- 
gation (GaBP) [131 HH HO] as well as numerous ap- 
plications [HI [21 H] ■ The best known sufficient condi- 
tion for its convergence is the walk-summable condi- 
tion [6l [7] (see also [3 Hj), which also provides new 
insights into the algorithm by interpreting it as com- 
puting weighted sums of walks (walk-sums) within the 
graph. Our aim in this present paper is to extend 
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this graphical/combinatorial view of GaBP to include 
estimation of the determinant (partition function) of 
the Gaussian graphical model. This work is also in- 
spired by the loop-series correction method for belief 
propagation [4j that was recently extended to Gaus- 
sian graphical models [5] . 

Our present study leads to a new perspective on GaBP 
having close ties to graphical zeta functions |T5]. We 
find that for walk-summable models the determinant 
may be represented as a product over all orbits (cyclic 
walks) of the graph. The estimate of the determi- 
nant provided by GaBP only captures totally back- 
tracking orbits, which can be embedded as orbits in 
the computation tree (universal cover) of the graph. 
The missing orbits may then be grouped into equiv- 
alence classes corresponding to backtrackless orbits. 
The orbit-product over each such equivalence class 
may be simply computed from the solution of GaBP. 
Also, the product over all backtrackless orbits may be 
interpreted as the determinant of a backtrackless adja- 
cency matrix of the graph with appropriately defined 
edge weights based on the GaBP solution. Finally, we 
propose a simple, efficient method to compute trun- 
cated orbit-products including all orbits up to some 
specified length and provide an error-bound on the re- 
sulting estimates. In certain classes of graphs (e.g., 
grids), this leads to an efficient method with complex- 
ity linear in the number of nodes and the required 
precision of the determinant estimate. 

This paper differs fundamentally from ^ in that we 
rely heavily on the walk-summable property to develop 
multiplicative expansions using infinitely many orbits 
of the graph, whereas develops additive expansions 
over a finite number of "generalized loops" (which may 
be disconnected) using methods of Grassman calcu- 
lus. Our present approach leads naturally to approx- 
imation methods (with accuracy guarantees in walk- 
summable models) based on truncated orbit-products. 
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2. Preliminaries 

2.1. Walks and Orbits of a Graph 

Let G be a graph on vertices (nodes) V = {1, . . . , n} 
with undirected edges € G. We may also 

treat each undirected edge as a symmetric pair 

of directed edges (ij) and (ji). A walk w is a se- 
quence of adjacent vertices {wo...wl) {wt G V for 
i = 0, . . . , L and {wf , Wf+i} e G for t = 1, .... L — 1) 
where \w\ = L is the length of the walk. A walk 
may be equivalently specified as a sequence of steps 
w = {{woWi){wiW2) ■ ■ ■ (wl^iWl)) such that each step 
is a (directed) edge of the graph that ends where the 
following step begins. A walk may visit the same node 
or cross the same edge multiple times and may also 
backtrack — that is, it may step back to the preceding 
vertex. A walk is closed if it begins and ends at the 
same node wq — wl- A closed walk is primitive if it 
is not a multiple of some shorter walk (e.g., the walk 
(1231) is primitive but (1231231) is not). We define an 
orbit £ = [w] to be an equivalence class of closed prim- 
itive walks, where two walks are considered equivalent 
[w] = [w'] if one is a cyclic shift of the other (i.e., if 
Wf = w' , , n for some s and t = 1, . . . , L). Hence, 
there is a one-to-one correspondence between orbits 
and (non-terminating) cyclic walks. 

The following classification of walks and orbits plays 
an essential role in our analysis: A walk (or orbit) is 
said to be reducible if it contains a backtracking pair of 
consecutive steps . . . {ij){ji) ■ ■ ■ , otherwise the walk is 
irreducible. By repeatedly deleting backtracking pairs 
until none remain one obtains the (unique) irreducible 
core 7 — T{w) of the walk w. For closed walks it may 
happen that 7 = 0, where = () denotes the trivial 
(empty) walk. We then say that the walk is totally 
reducible. We say that a walk is non-trivial if it is not 
totally reducible. Totally reducible walks have been 
called backtracking |7j, although totally backtracking 
is perhaps a better description. Irreducible walks have 
been called backtrackless (or non-backtracking) else- 
where in the literature. 

Notation. C denotes the set of all orbits of G, T{C) 
denotes the irreducible (backtrackless) orbits, and we 
partition C into (disjoint) equivalence classes £^ = 
{£ e C\T{£) = 7} for 7 e r(£). In particular, £0 
denotes the class of totally backtracking orbits, which 
plays a special role in our interpretation of GaBP. We 
will use £ to denote a generic orbit and reserve 7 to 
denote irreducible orbits. 

Example. Orbits [1231], [1231451], [123421561] are 
backtrackless; [1234321], [1232421], [1231321] are to- 
tally backtracking; [123241] is both reducible and non- 



trivial (neither backtrackless nor totally backtracking). 

2.2. Gaussian Belief Propagation 

A Gaussian graphical model is a probability distribu- 
tion 

p{x) ^ Z^^exp{-^x'^Jx + h^x} (1) 

of random variables x € M" where J is a sparse, sym- 
metric, positive-definite matrix. The fill-pattern of J 
defines a graph G with vertices V = {1, . . . ,n} and 
edges (ij) for all 7^ 0. The partition function 
is defined by Z{h,J) = J exp{— ^a;^ -I- h"^x}dx = 
[(27r)" det J-^] ^.'^h'^J'^h normalize the dis- 

tribution. Given such a model, we may compute the 
mean vector /i = J p[x)xdx = J~^h and covariance 
matrix K ^ J p{x){x ~ ^){x — dx = J^^ . This 
generally requires 0{n^) computation in dense graphs 
using Gaussian elimination. If G is sparse and only 
certain elements of K are required (the diagonal and 
edge- wise covariances) , then the complexity of Gaus- 
sian elimination may be substantially reduced (e.g., 
C'(n'^/^) in planar graphs using nested dissection) but 
still generally has complexity growing as 0{w'^) in the 
tree-width w of the graph. 

Gaussian belief propagation (GaBP) is a simple, dis- 
tributed, iterative message-passing algorithm to es- 
timate the marginal distribution p{xi) of each vari- 
able, which is specified by its mean fii and variance 
Kii. GaBP is parameterized by a set of messages 
mij{xj) = ea'^'j^j+'^'j^i defined on each directed edge 
(ij) of the graph {mij is regarded as a message being 
passed from i to j). The GaBP equations are: 




where ipi ^ e 2-''"^«+'^'^% i/j^j = (,~Jijx^xj g^^^ g- 
notes the set of neighbors of i in G. This reduces to 
the following rules for computing (a, /3) -messages: 

Pi] = -Jij{Jii-ai\jy^(hi+Pi\j) (2) 

where a,\j = Efcea*\i and Ay = T.k^di\o Pk^- 
These equations are solved by iteratively recomputing 
each message from the other messages until conver- 
gence. The marginal distribution is then estimated as 
p^P(a;i) (X ■il^iixijYlj.^Q^mkiixi), which gives variance 
estimates K'°'^ = {Ju — aki)~^ and mean estimates 
— K^^{hi -f J^kPk-i). In trees, this method is 
equivalent to Gaussian elimination, terminates after 
a finite number of steps and then provides the cor- 
rect marginals. In loopy graphs, it may be viewed as 
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performing Gaussian elimination in the computation 
tree (universal cover) of the graph 10, 7 (obtained by 
"unrolling" loops) and may therefore fail to converge 
due to the infinite extent of the computation tree. If 
it does converge, the mean estimates are still correct 
but the variances are only approximate. We also ob- 
tain an estimate of the pairwise covariance matrix on 
edges {i,j} £ G: 

-1 



Ji' 



J, 



In this paper, we are concerned with the GaBP esti- 
mate of the determinant Z = det K — det J"-"^ (which 
is closely linked to computation of the partition func- 
tion Z). We obtain an estimate of Z from the GaBP 
solution as: 



^bp 



w^r n 

{ij}eG 



7bp 



(3) 



where Z'^p = k""^ and z'^f = deiX'?'^.,. The mo- 
tivation for this form of estimate is that it becomes 
exact if G is a tree. In loopy graphs, there may gen- 
erally be no stable solution to GaBP (or multiple un- 
stable solutions). The main objective of this paper is 
to interpret the estimate Z'^p in the context of walk- 
summable models (described below), for which there is 
a well-defined stable solution, and to suggest methods 
to correct this estimate. Note that the variance and 
covariance estimates (and hence the determinant esti- 
mate Z^P) are independent of h and the /3-messages 
in (12), they are determined solely by the a-messages 
determined by J. Since GaBP correctly computes the 
means in walk-summable models, we are mainly con- 
cerned with how to correct Z^^ (and hence its deriva- 
tives, which correspond to the GaBP estimates of vari- 
ances / covariances) . 

Walk-Sum Interpretation Our approach in this 
paper may be considered as as extension of the walk- 
sum interpretation of GaBP '7\. Let J be normalized 
to have unit-diagonal, such that J = I — R with R 
having zeros along its diagonal. The walk-sum idea is 
based on the series K — {I — R)~^ — J^k ^'^^ which 
converges if p{R) < 1 where p{R) denotes the spectral 
radius of the matrix R (the maximum modulus of the 
eigenvalues of R). This allows us to interpret Kij as 
a sum over all walks in the graph G which begin at 
node i and end at node j where the weight of a walk is 

defined as i?*^' = n(y)eu) '"I^'"^™^ ^^'^ "ij (^) is a count 
of how many times step (ij) occurs in the walk. We 
write this walk-sum as Kij = X^uj i^j ■ However, 
in order for the walk-sum to be well-defined, it must 
converge to the same value regardless of the order in 



which we add the walks. This is equivalent to requir- 
ing that it converges absolutely. Thus, we say that 
R is walk-summable if X^Mi i-^j \R^\ converges for all 
i,j € V. This is equivalent to the spectral condition 
that /c(|-R|) < 1 where \R\ = {\rij\) is the element- 
wise absolute-value matrix of R. A number of other 
equivalent or sufhcient conditions are given in . 

In walk-summable models it then holds that variances 
correspond to closed walk-sums Ku = » 
means correspond to a (reweighted) walk-sum over all 
walks which end at a specific node fii = J2w*^i ^*^™ 
(here * denotes the arbitrary starting point of the 
walk). Moreover, we may interpret the GaBP message 
parameters (a, /?) as recursively computing walk-sums 
within the computation tree [7]. This implies that 
GaBP converges in walk-summable models and con- 
verges to the same "walk-sum" solution independent 
of the order in which we update messages. This inter- 
pretation also shows that GaBP computes the correct 
walk-sums for the means but only computes a subset 
of the closed walks needed for the variances. Specifi- 
cally, k]"^ only includes totally backtracking walks at 
node i. This is seen as a walk is totally backtracking if 
and only if it can be embedded as a closed walk in the 
computation tree of the graph and it is these closed 
walks of the computation tree which GaBP captures 
in its variance estimates. 

3. Orbit-Product Interpretation of 
Gaussian BP 

3.1. Determinant Z as Orbit-Product 

Let Z{R) = det(/ — R)~^. In walk-summable mod- 
els, we may give this determinant another graphical 
interpretation as a product over orbits of a graph, one 
closely related to the so-called zeta function of a graph 

[El- 
Theorem 1 If p{\R\) < 1 then it holds that Z{R) = 
Y\i{^ — R^)^^ = where the product is taken over 

all orbits of G and R^ = Y[(ij)ee^7j^^^^ where nij{£) is 
the number of times step (ij) occurs in orbit £. 



Proof logdet(J - i?)-i = trlog(/ - R)~^ = tr^ifc ^ 

-R" _ Y^oo (-R™)*" _ 



_ V 

-/closcdii? \w\ ^primitive Z—^m—l m|tu| 

E primitive ™ R log (1 - R^^T^ = Eorbits£l0g(l - 

i?^)-i = logn<?(l - i?^)-i. We have used the iden- 
tity log det A — ti log A and the series expansion 
log(/ — A)~^ — J2kLi 4"- Ea-ch closed walk is ex- 
pressed as a multiple of a primitive walk. Every prim- 
itive walk w has exactly |w;| distinct cyclic shifts, o 
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We emphasize that p(|i?|) < 1 is necessary to in- 
sure that the the orbit-products we consider are weU- 
defined. This condition is assumed throughout the re- 
mainder of the paper. 

3.2. Z^^ as Totally Backtracking Orbits 

Totally backtracking walks play an important role in 
the walk-sum interpretation of the GaBP variance es- 
timates. We now derive an analogous interpretation of 
Z^"^ defined by ©: 

Theorem 2 Z'^p — Wi^^^^ Zi where the product is 
taken over the set of totally backtracking orbits of G. 

Although this result seems intuitive in view of prior 
work, its proof is non-trivial involving arguments not 
used previously. To prove the theorem, we first sum- 
marize some useful lemmas. Consider a block ma- 
trix A — (^ii/li2; j421j422). The Schur complement 
of block All is ^22 = A22 - A2iA^^A2i. It holds 
that det^ = det^iidetA22 and (^22)^^ = (^"^)22- 
Using these well-known identities, it follows: 

Lemma 1 Let R — {R11R12', R21R22) and K = {I ~ 
= {KiiKi2;K2iK22). Then detKn ^ 



Z(R.22) ■ 



For walk-summable models, we then have 



det Kii = 



n 



^£zG I £ intersects Gi 



where the final orbit-product is taken over all orbits 
of G which include any node of subgraph Gi (corre- 
sponding to submatrix Rn). Next, using this result 
and the interpretation of GaBP as inference on the 
computation tree, we are led to the following inter- 
pretation of the quantities Z'"'^ and Z^^ appearing in 
^ . Let Ti denote the computation tree of the graph 
G with one copy of node i marked. Let Ty denote 
the computation tree with one copy of edge {«, j} G G 
marked. Then, 

Lemma 2 z}''^ = Y[eeT \i£e where the product is 
over all orbits of Ti that include the marked node i. 

~ Y[eeT -\i£eoTjee where the product is over all 
orbits of Tij that include either endpoint of the marked 
edge {ij}. 

Proof of Theorem 2. Using Lemma 2 and the cor- 
respondence between orbits of the computation tree 
and totally backtracking orbits of G, we may expand 
^ to express Z^^ entirely as a product over totally 
backtracking orbits Z^p = Hiec ^i' where is 



the count of how many times i appears in the orbit- 
product — the number of times it appears in the numer- 
ator of ^ minus the number of times in appears in 
the denominator. It remains to show that Ng = 1 for 
each totally backtracking orbit. This may be seen by 
considering the subtree Tg of the computation tree T 
traced out by orbit £. Let v and e respectively denote 
the number of nodes and edges of (hence, e — v—1) 
and let c denote the number of edges of T with exactly 
one endpoint in . First, we count how many powers 
of Zi appear in the orbit product . For each 

vertex i d Ti we may pick this as the marked node 
in the computation tree and this shows one way that 
£ can be embedded in Ti so as to include its marked 
node. Thus, v gives the total number of multiples 
of Zg in Yli ■ Similarly, we could mark any edge 
{i,j} G T with one or both endpoints in Tg and this 
gives one way to embed £ into Tij so as to intersect the 
marked edge. Thus, the product JJ-j Z^J contributes 

e -I- c powers of Zt. Lastly, the product Oy 
contains 2e -I- c powers of Zg. This represents the num- 
ber of ways we may pick a directed edge {if) of T such 
that at least one endpoint is in Tg. The total count is 
then iV^ = w + (e + c) — (2e + c) = — e = 1. o 

Combining Theorems 1 and 2, we obtain the following 
orbit-product correction to Z'^P; 

Corollary 1 Z = Z'^p x H^^c, 

This formula includes a correction for every missing 
orbit, that is, for every non-trivial orbit. This implies 
that Z = Z'^P for trees since all orbits of trees are 
totally backtracking. 

3.3. Z'^P Error Bound 

One useful consequence of the orbit-product interpre- 
tation of Z^P is that it provides a simple error bound 
on GaBP. Let g denote the girth of the graph G, de- 
fined as the length of the shortest cycle of G. We note 
that the missing orbits I ^ L% must all have length 
greater than or equal to g. Then, 



Theorem 3 



log V 



< 



9(i~p(ifli)) ■ 



Proof. 

h Z I 



We derive the chain of inequalities: 

(b) 

e\>g\"-"^^''' ^ 



Et^c^ogZt 



< 



El. 



logZ, 



E|£|>ffl0g(l 
np" v^oo I 

g l^k=OP 


-\R\')-' = 






• (a) 


\\ogZe\ = 


2^k=l k 






(c) The proof of 


that trEfc>i4- - E^log(l - 



£_ 

=g fc 



< 



(b) 



log(l 



Similarly, 
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trE.>,^ = E|,|>,log(l-|i?n-^-0 

This is consistent with the usual intuition that behef 
propagation is most accurate in large girth graphs with 
weak interactions. 

4. Backtrackless Orbit Correction 

In this section we show that the set of orbits omitted 
in the GaBP estimate can be grouped into equivalence 
classes corresponding to backtrackless orbits and that 
the orbit-product over each such equivalence class is 
simply computed with the aid of the GaBP solution: 

Theorem A: Z = Z^^ x 117^^0 -^7 where the product is 
over all backtrackless orbits of G and we define 

(jj)e7 

where = and ai\j = J2kedi\j o'ki is com- 

puted from the solution of GaBP. 

In comparison to Corollary 1, here the correction fac- 
tor is expressed as an orbit-product over just the back- 
trackless orbits (whereas Corollary 1 uses a separate 
correction for each non-trivial orbit). However, all or- 
bits are still correctly accounted for because we modify 
the edge weights of the graph so as to include a fac- 
tor (1 — ai\j)~^ (computed by GaBP) which serves 
to "factor in" totally-backtracking excursions at each 
point along the backtrackless orbit, thereby generating 
all non- trivial orbits. 

The basic idea underlying this construction is depicted 
in Figure 1. For each backtrackless orbit 7 we de- 
fine an associated computation graph G7 as follows. 
First, we start with a single directed cycle based on 
7 = [7172 ■ ■ ■ 7l] (any duplicated nodes of the orbit 
map to distinct nodes in this directed cycle). Then, 
for each node jk of this graph, we attach a copy of the 
computation tree TJ^\-fk+l^ obtained by taking the full 
computation tree T-y^. rooted at node 7^ and deleting 
the branch (7^,7^+1) incident to the root. This con- 
struction is illustrated in Figure l(a,b) for the graph 
G = K4 and orbit 7 = [(12) (23) (31)]. The cycle has 
"one-way" directed edges whereas each computation 
tree has "two-way" undirected edges. This is under- 
stood to mean that walks are allowed to backtrack 
within the computation tree but not within the cycle. 
The importance of this graph is based on the following 
lemma (the proof is omitted) : 

Lemma 3 Let j be a backtrackless orbit of G. Then, 
there is a one-to-one correspondence between the class 
of orbits £7 of G and the non-trivial orbits of Gj. 




[(12)(23)(31)] ^31 




«1\2 — «31 + «41 a2\3 = «12 + «42 a3\l = "23 + «43 



CF^ *-0 

Figure 1. Illustration of construction to combine equivalent 
orbits, (a) The graph G = K4. (b) The computation graph 
G7 for 7 = [(12)(23)(31)]. (c) Finite graph with self-loops 
at each node to capture totally backtracking walks, (d) 
Equivalent graph with modified edge weights to capture 
totally backtracking walks. 

Next, we demonstrate how to compute all of the orbits 
within an equivalence class as a simple determinant 
calculation based on the backtrackless orbit 7 and the 
GaBP solution. Let R!y be defined as the edge-weight 
matrix of a simple single-loop graph based on 7 with 
edge- weights defined by r' ^ = , _"''"^''+i — . This 

construction is illustrated in Figure 1(d). Then, 
Lemma 4 = det(/ - B!^y = U^^^^ Z^. 

Proof. Using Lemma 3, we see that the orbit-product 

n^e^ equal to the product over all non-trivial 

orbits of the graph G-y, that is, the product over all or- 
bits in G7 which intersect the subgraph corresponding 
to 7. Using Lemma 1, this is equivalent to comput- 
ing the determinant of the corresponding submatrix 
of Kg^ = (/ — Rgi)~^ where Rcg is the edge- weight 
matrix of the computation graph. This is equivalent 
to first eliminating each computation tree (by Gaus- 
sian elimination/GaBP) attached to each node of 7 
to obtain a reduced graphical model I — Rj and then 
computing det(/ — R^)~^ . Using the GaBP solution, 
the effect of eliminating each computation tree is to 
add a "self-loop" (diagonal element) to R-^, with edge- 
weight a^^\7^+i = E„^7^+i a«,7/=' obtained by sum- 
ming the incoming messages to node 7/5 from each 
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of its neighbors in the subtree Tj^\j^_^^. This ehm- 
ination step is illustrated if Figure l(b,c). We may 
use the orbit-product formula to compute the deter- 
minant. However, there are infinitely many orbits in 
this graph due to the presence of a self-loop at each 
of the remaining nodes. At each node, an orbit may 
execute any number of steps m around this self-loop 
each with edge-weight aj^\j^^-^. Summing these, we 

obtain E™=o"Ta7<=+i = (1 " "7A7.+i)"^- Hence, 
we can delete each self-loop and multiply the follow- 
ing edge's weight by (1 — o;j^\j^^-^)'^ and this pre- 
serves the value of the determinant. This final reduc- 
tion step is illustrated in Figure l(c,d). Then, the 
orbit-product H^ec equal to det(/— i?^^)"^ (e.g., 

based on the graph seen in Figure 1(d)). It is straight- 
forward to compute the resulting determinant with re- 
spect to the single (directed) cycle graph with edge- 
weights Riy. There is only one orbit in this graph and 



O- 



o- 



-o- 



-o- 



-o 




(a) O O O 

Figure 2. (a) 3 x 3 grid G. (b) Graph G' representing the 
backtrackless adjacency matrix R' . Each node ij repre- 
sents a directed edge of G, directed edges are drawn be- 
tween nodes ij and jk which are non-backtracking [k ^ i). 



Before providing the proof, we establish that 
walk-summability with respect to R implies walk- 
summability with respect to R': 



hence det(/ — R' 



(1 - {R'yy^ where Lemma 5 If p{\R\) < 1 then p{\R'\) < pi\R\). 



(RV = n(,,)6^«,)"-^^^^ and rj^ = r,,(l - a,\^)-\ o 

Proof of Theorem 4- Using these results, it is now 
simple to show ^ = Ui^c^ = = 

n ^0 det(/ - i?' )-i = n ^0(1 - {B!Vry o 



5. Backtrackless Determinant 
Correction 

Next, we show that the correction factor 

^=n^;=ndet(/-i?;)-^ 



7#0 



7#0 



may also be calculated as a single determinant based 
on the following backtrackless adjacency matrix of the 
graph. We define R' G m2|g|x2|g| foUows. Let the 
rows and columns of R' be indexed by directed edges 
{ij) of the graph G. Then, the elements of R' are 
defined 



R 



0, 



j — k and i 7^ I 
otherwise. 



(4) 



This construction is illustrated in Figure 2. Note that 
the walks generated by taking powers R' correspond to 
backtrackless walks of the graph G. The weight of an 
edge {{ij){jk)) in R' is defined as the (modified) edge- 
weight r'jj^ of the endpoint (jk). The weight of an orbit 
in R' may then be equivalently defined as the product 
of node weights r'^^ taken over the orbit in i?', which is 
equal to the weight of the corresponding backtrackless 
orbit of G (using the modified edge weights r-^). 

Theorem 5 Z = Z^p x Z' where Z' ^ dct{I - R')-\ 
that is, det(/ - i?)-i = Z^P x dct(/ - R')-^ . 



Proof. Once the a-parameters converge, the /3- 
parameters follow a linear system /3k+i = R'Pk + b 
[S] . Hence, the asymptotic convergence rate of GaBP 
is p{R')- Compare this to the Gauss- Jacobi (GJ) iter- 
ation pk+i = iJ,k + {h- Juk) = Sf=o R*^^ (a*o = 0), 
which has convergence rate p{R)- It is clear that the 
GaBP iteration captures a superset of those walks 
computed by GJ at each iteration (because the depth-Zc 
computation tree includes all fc-length walks). Hence, 
for non-negative models (i? > and /i > 0) it must 
hold that the error in the GaBP estimate of p is less 
than or equal to the error of GJ (at every iteration). 
This implies p{R') < p{R) if i? > 0, from which we 
conclude p{\R'\) < p(|^|) hr walk-summable models, 
o 

Proof of Theorem 5. By construction, there is a one- 
to-one weight-preserving correspondence between or- 
bits of G" and backtrackless orbits of G. The result 
then follows from the orbit-product representation of 
Z' over G' (Theorem 1, Lemma 5), which is equivalent 
to the backtrackless orbit-product of Theorem 4. o 



One useful consequence of this result is that the error 
bound of Theorem 3 can be improved to ^ log ~ 

n 1^08^ I ^ S(l-P(|fl/|))- 

It is impractical to compute the complete correction 
factor Z' = det(/ — i?')^-'^, as this is not easier than di- 
rectly computing Z = det(/— i?)~^. However, because 
R' is itself walk-summable, we can use this representa- 
tion as a starting point for constructing approximate 
corrections such as the one considered in the next sec- 
tion. 
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6. Block-Resummation Method 

Next, we consider an efficient method to approxi- 
mate Z{A) = det{I — A)~^ for walk-summable models 
p(|v4|) < 1. This method can be used to either di- 
rectly approximate Z(R) {A = R) or to approximate 
the GaBP-correction Z' {A = R'). 

Given a graph G based on vertices V, we specify a set 
of blocks B = {Bk C y, fc = 1, • ■ • , \B\) chosen such 
that: (1) Every short orbit |£| < L is covered by some 
block B (zB, and (2) If B,B' e B then B n B' e B. 
We also define block weights as follows: wb — ^ 
for maximal blocks (not contained by another block) 
and Wb = 1 — X^bob^s' non-maximal blocks 
(these weights may be negative). This insures that 
Ssob'^'^s' ~ 1 each B £ B. Then, we define our 
estimate 

Zb = 1[ ZT - n(d^t(/ - AB)-^r- (5) 

B B 

where Ab denotes the \B\ x \B\ principle submatrix of 
A corresponding to B. 

This approximation method is similar in spirit to ap- 
proximations used elsewhere (e.g., Kikuchi approxima- 
tions to free-energy [H]). However, the new insights 
offered by the orbit-product view allows us to give our 
estimate a precise interpretation in walk-summable 
Gaussian models: 

Theorem 6 = JlfeCB where Cb = liBeB^B 
and Cb is the set of all orbits covered by B. 

Proof, zb = UBUieBzr = n.e^. ^F""^ = 

n^erB Z^ where J^bdi'^b — 1 follows from the defi- 
nition of the block weights, o 

Moreover, we can then bound the error of the estimate. 
Noting that Cb includes all short orbits of the graph, 
we can derive the following result by a similar proof as 
for Theorem 3: 

Corollary 2 i | fog ^ | < ^^fMJl_ . 

Thus, for the class of models with p < 1, we obtain an 
approximation scheme which converges to the correct 
determinant as the parameter L is made large with er- 
ror decaying exponentially in L. The estimate Zb(R) 
includes all orbits that are covered by some block. The 
improved GaBP-based estimate Z^pZb{R') includes 
all orbits £ such that j = T{£) is covered by some block. 
Thus, the GaBP-based correction includes many more 
orbits. We also note that the error-bound using the 
GaBP-based estimate is typically smaller as we have 
shown that p(|i?'|) < p{\R\) (if p(|i?|) < 1). 




Figure 3. Demonstration of determinant approximation 
method for 256 x 256 periodic grid with uniform edge 
weights r £ (0, .25). Plots of (a) p{\R\) and p{\R'\) vs 
r; (b) (ilog of) Z, Z^^ and Zb (with L = 2,4,8,16,32) 
vs r; (c)" i log(Z'^PZ^) vs r; and (d) ^\log{Z'^ Zb)\ and 
i| log(Z-^Z''PZB)| vs i = 2, 4, 8, 16, 32 for r = .23. In (b) 
and (c) the estimates for L — 8, 16, 32 are all nearly exact 
(and therefore hard to distinguish in the plot) and the er- 
rors are largest near the walk-summable threshold r = .25. 
Estimates (b) are not based on GaBP and are actually 
worst than Z^'^ for L — 2. However, the GaBP-corrections 
(c) are strictly better than Z^'^. 

Construction of B for Grids To achieve an error 
bound ijlog^l < e we must choose L ~ loge~^. 
Then, the computation needed to achieve this preci- 
sion will depend on both the number of blocks and the 
block size needed to cover all orbits up to this length. 
In certain classes of sparse graphs, it should be pos- 
sible to control the complexity of the method. As an 
example, we demonstrate how to choose blocks for 2D 
grids. Consider the ^/nx ^Jn square grid in which each 
vertex is connected to its four nearest neighbors. We 
may cover this graph by L x L blocks shifted (both 
vertically and horizontally) in increments of (let L 
be even). It can be seen that this set of blocks covers 
all loops shorter than L. To include all intersections 
of blocks, we add L x -j, -I x L and -j x -j blocks. The 
block weights are wlxl = 1, = '«^l/2xl = -1 

and Wi/2xL/2 — 1- The complexity of computing the 
determinant of an L x L block is OiL?") and the total 
number of blocks is Oinjl?). Hence, the total com- 
plexity is 0{nL) — ©(nloge^^). 

We test our approach numerically on a 256 x 256 square 
grid (with periodic boundary conditions). We set all 
edge weights to r and test the quality of approximation 
using both estimates Zb{R) and Z"""^ Zb{R') for r E 
(0, .25) [J = I — R becomes indefinite for larger values 
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of r) and block sizes L = 2,4,8,16,32. The results 
are shown in Figure 3. As expected, accuracy rapidly 
improves with increasing L in both methods and the 
GaBP-correction approach is more accurate. 

7. Conclusion and Future Work 

We have demonstrated an orbit-product representa- 
tion of the determinant (the partition function of the 
Gaussian model) and interpreted the estimate ob- 
tained by GaBP as corresponding to totally back- 
tracking orbits. Furthermore, we have shown how to 
correct the GaBP estimate in various ways which in- 
volve incorporating backtrackless orbits (e.g. cycles) 
of the graph. In particular, we demonstrated an ef- 
ficient block-resummation method to compute trun- 
cated orbit-products in sparse graphs (demonstrated 
for grids). These methods also extend to address es- 
timation of the matrix inverse (the covariance matrix 
of the Gaussian model), which may in turn be used 
as an efficient preconditioner for iterative solution of 
linear systems. We leave these extensions for a longer 
report. 

In future work, we plan to extend the method of con- 
structing an efficient set of blocks to other classes 
of sparse graphs. It may also be fruitful to extend 
our analysis to generalized belief propagation [14] in 
Gaussian models. In a related direction, we intend 
to explore methods to "bootstrap" GaBP using the 

factorization Z{R) = (jlj^o ^(--R^"))) ' which fol- 
lows from the formula {I — R)^^ — nfe(-^ + i?^''). By 
computing Z^^{—R^ ) for small values of k we may 
capture short backtrackless orbits of the graph. An- 
other direction is to investigate generalization of the 
formula Z = Z^^Z' to non-walksummable models, 
perhaps using methods of [3]. A related idea is to 
approximate a non-walksummable model by a walk- 
summable one and then correct estimates obtained 
from the walk-summable model to better approximate 
the non-walksummable model. 
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